


Institutional Archive of the Naval Postgraduate School 





Calhoun: The NPS Institutional Archive 
DSpace Repository 


Theses and Dissertations 1. Thesis and Dissertation Collection, all items 


1997-03 


A methodology for improving the usability of 
the ANVIS/HUD computer based trainer 


Rozelle, Daniel R 


Monterey, California. Naval Postgraduate School 


http://ndl.handle.net/10945/26066 


Downloaded from NPS Archive: Calhoun 


Calhoun is the Naval Postgraduate School's public access digital repository for 


f (8 D U DLEY research materials and institutional publications created by the NPS community. 
«ist : Calhoun is named for Professor of Mathematics Guy K. Calhoun, NPS's first 


NY KNOX appointed — and published — scholarly author. 


LIBRARY Dudley Knox Library / Naval Postgraduate School 
411 Dyer Road / 1 University Circle 
Monterey, California USA 93943 





http://www.nps.edu/library 


NPS ARCHIVE 


1997,03 
ROZELLE, D. 


NAVAL POSTGRADUATE SCHOOL 
MONTEREY, CALIFORNIA 
















A METHODOLOGY FOR IMPROVING THE 
USABILITY OF THE ANVIS/HUD COMPUTER 
BASED TRAINER 


by 
Daniel R. Rozelle 


March 1997 





Principal Advisor: Anthony Ciavarelli J 


Approved for public release; distribution is unlimited 


'Tnesis 


R824 





- NOX } {aay 
 DaPF% Col 5: 2 


My) Ls : — TN 
} i’ % f et 


fe = » «Nias § ey C * 


DUDLEY KNOX LIBRARY 
NAVAL POSTGRADUATE SCHOOL 
MONTEREY, CA 93943-5101 


: = ———< —— 
= _ —_ a = = = as 


REPORT DOCUMENTATION PAGE (| aesat cue om 


Public reporting burden for this collection of information 1s estumated to average ] hour per response, including the ume for reviewing instruction, searching existing data sources, | 
gathenng and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this 
cullectun of mformatrun, nicloding suggestions for redacing thas Gurden, tc Washington Licadquarters Services, Directorate for dunmmation Operations aid Reports, 1215 Setersun 
Davis Highway, Suite 1204, Arlington, VA 22202-4302, and to the Office of Management and Budget, Paperwork Reduction Project (0704-0188) Washington DC 20503 


1. AGENCY USE ONLY (eave blank) 2 REPORT DATE 3. REPORT TYPE AND DATES COVERED 
March 1997 Master’s Thesis 
4. TITLE AND SUBTITLE: A METHODOLOGY FOR >. FUNDING NUMBERS 
IMPROVING THE USABILITY OF THE ANVIS/HUD | 
COMPUTER BASED TRAINER 


6. AUTHOR(S) Rozelle, Daniel R. 


7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) . PERFORMING 
Naval Postgraduate School ORGANIZATION 
Monterey, CA 93943-5000 REPORT NUMBER 









11. SUPPLEMENTARY NOTES The views expressed in this thesis are those of the author and do not reflect the 
official policy or position of the Department of Defense or the U.S. Government. 
12a. DISTRIBUTION/AVAILABILITY STATEMENT: Approved for public release; } 12b. DISTRIBUTION CODE: A 
distribution is unlimited. 


113. ABSTRACT (maximum 200 words) ? | 





Computer software has taken an increasingly larger role in the U.S. Navy. It is used in nearly every 
| 


facet of naval operations, from administrative chores to controlling complex weapons systems. Because 
of the high cost of software and the potential for inadvertent misuse, it is umportant that software be 
easy to use and understand. This thesis explores the methods and techniques available for conducting 
software usability evaluations. Using one of the methods described in this thesis, actual software 
usability testing is done on a recently developed computer-based tramimg (CBT) program. The CBT 
program evaluated in this study is designed to instruct helicopter pilots in the use of the AN/AVS-7 | 
ANVIS/HUD. The device is an advanced night vision goggle system that is comprised of the AN/AVS- 

6 ANVIS night vision goggle (NVG) set and a Heads-Up Display (HUD). This thesis describes the \ 
usability test conducted on the ANVIS/HUD CBT and establishes a methodology that can be used, not 
only on future versions of the ANVIS/HUD CBT, but on other PC oriented software. The result of this 
usability test show that improvement can be made to the navigation methud used by the CBT and the 
presentation of instructional material. 


14. SUBJECT TERMS Usability Testing, Computer Based Training, Night Vision |!>- NUMBER OF 
Goggles PAGES 108 


LOmePRICE CODE 
17. SECURITY CLASSIFICA- | 18. SECURITY CLASSIFI- = 1omSECURTT CLASSIFICA-* 1/20. “IMT A MON OF 


TION OF REPORT CATION OF THIS PAGE TION OF ABSTRACT ABSTRACT 
Unclassified Unclassified Unclassified UL 
NSN 7540-01-280-5500 Standard Form 298 (Rev. 2-89) 


Prescribed by ANSI Std. 239-18 298-102 





Approved for public release; distribution is unlimited 


A METHODOLOGY FOR IMPROVING THE USABILITY OF THE 
ANVIS/HUD COMPUTER BASED TRAINER 


Daniel R. Rozelle 
Lieutenant Commander; United States Navy 
B.A., Findlay College, 1985 


Submitted in partial fulfillment of the 
requirements for the degree of 


MASTER OF SCIENCE IN 
INFORMATION TECHNOLOGY MANAGEMENT 
from the 


NAVAL POSTGRADUATE SCHOOL 
March 1997 


DUDLEY KNOX LIBRARY 
NAVAL POSTGRADUATE SCHOOL 
MONTEREY, CA 93943-5101 


ABSTRACT 


Computer software has taken an increasingly larger role in the U.S. Navy. 
It is used in nearly every facet of naval operations, from administrative chores to 
controlling complex weapons systems. Because of the high cost of software and 
the potential for inadvertent misuse, it is important that software be easy to use and 
understand. This thesis explores the methods and techniques available for 
conducting software usability evaluations. Using one of the methods described in 
this thesis, actual software usability testing is done on a recently developed 
computer-based training (CBT) program. The CBT program evaluated in this 
study is designed to instruct helicopter pilots in the use of the AN/AVS-7 
ANVIS/HUD. The device is an advanced night vision goggle system that is 
comprised of the AN/AVS-6 ANVIS night vision goggle (NVG) set and a Heads- 
Up Display (HUD). This thesis describes the usability test conducted on the 
ANVIS/HUD CBT and establishes a methodology that can be used, not only on 
future versions of the ANVIS/HUD CBT, but on other PC oriented software. The 
result of this usability test show that improvement can be made to the navigation 


method used by the CBT and the presentation of instructional material. 
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I. INTRODUCTION 


A. PURPOSE 

It is the intent of this thesis to develop and apply a methodology that can be 
used to assess the usability of the current and future versions of the AN/AVS-7 
Aviators Night Vision Imaging System/Heads-Up Display (ANVIS/HUD) 
Computer Based Trainer (CBT). As the Navy, through the efforts of the Naval 
Aviation Safety School and the Naval Postgraduate School (NPS), moves toward 
actual use of the ANVIS/HUD software, it is important that the product not only 
meet instructional requirements, but those for ease of use as well. We must not 
fail to recognize the needs of the military aviator to have an intuitive, easy to use 
tool for learning. Otherwise, the risk of investing in, creating and relying on a 
method of instruction that may be considered too difficult, time consuming or 


distracting is created. 


B. OBJECTIVE 

In order to achieve the stated purpose of this thesis three specific objectives 
were accomplished. First, a selected review of the related literature on software 
usability evaluation was conducted. This literature review yielded an 
understanding of the most effective and insightful methods of usability evaluation. 
Second, using the knowledge from the literature review, a methodology was 
developed for usability testing and applied in the assessment of the ANVIS/HUD 
CBT. In addition to the formal investigation of usability, input from helicopter 
aviators who have used the ANVIS/HUD CBT was solicited. Finally, the results 
of the usability evaluation were analyzed and summarized in order to capture the 


benefits of this study for future ANVIS/HUD CBT development. 


C. BACKGROUND 

I The AN/AVS-7 ANVIS/HUD 

The AN/AVS-7 ANVIS/HUD is an advanced electro-optical system that 
serves as an aid to rotary-wing pilots and copilots for night flying. This system 1s 
a combination of the AN/AVS-6 ANVIS Night Vision Goggle (NVG) set and a 
Heads-Up Display (HUD). It will ultimately be installed in most of the Navy’s 
1553 data-bus configured helicopters. The ANVIS is an electro-optical image 
intensifier system designed to provide aviators with the optimum capability to see 
in the dark and perform nap-of-the-earth and other terrain flight modes during 
starlight conditions. With the integrated HUD on the AN/AVS-7 ANVIS/HUD, 
the pilot and copilot can observe aircraft flight data without having to frequently 
look down at the flight instrument panel. Information from onboard sensors is 
displayed as symbology that is superimposed within the ANVIS might vision 
goggle field of view. The availability of HUD symbology, which depicts such 
flight and system information as altitude, heading, airspeed, velocity vector, engine 
status and warning signals, provides essential information from the pilot’s normal 
instrument scan. This helps to reduce crew fatigue and allows the pilot and copilot 
to maintain an aggressive out-of-cockpit visual scan for improved flight monitoring 
and situation awareness. 

De Computer Based Training for Night Vision Goggles 

In 1992, out of concern for the number of accidents involving NVG flight, 
Naval Aviation School faculty from NPS participated in a comprehensive study of 
night vision device training. Results of the study showed that improvements 
could be made in night vision device training in the areas of improved standard 
operating procedures, revisions to training doctrine and enhancements to training 
equipment and simulation devices (Ciavarelli, Sengupta and Baer, 1994). To that 


end, the Naval Aviation Safety School is in an agreement with Naval Air Systems 


Command (NAVAIRSYSCOM) PMA-205 (Training Systems) to develop a CBT 
application for the AN/AVS-7 ANVIS/HUD unit for naval rotary-wing aircraft. A 
number of studies have now been completed on the use of CBTs and multimedia 
to train pilots in the use of NVGs. Most recently the ANVIS/HUD, in pursuit of 
that goal. Initial work (Bryant and Day, 1994) provided a prototype interactive 
CBT that incorporated sound, video and animation. Concurrently, Ciavarelli et 
al., (1994) conducted a study covering NVG training requirements and a review of 
the technologies available for improving NVG training. These works were 
followed by a closer look at learning theory and the appropriate use of specific 
media in presenting content to achieve desired learning objectives (Meza, 1995). 
A fourth study (Epperson, 1995) was completed that detailed the techniques and 
requirements for implementing animation within the NVG multimedia CBT 
system. The culmination of this effort to date is the interactive AN/AVS-7 
ANVIS/HUD CBT Program for the HH-60H helicopter developed by Kern and 
Shaffer (1996). This CBT program is a complete re-engineering of a commercially 
delivered computer-based ANVIS/HUD trainer originally designed for the UH-1N 
helicopter. By drawing on the earlier works mentioned above and using portions 
of code, graphics and text from the UH-IN ANVIS/HUD CBT, a thoroughly 
revised and updated CBT was produced. 

3. The HH-60H AN/AVS-7 ANVIS/HUD CBT 

The new HH-60H version of the ANVIS/HUD trainer 1s composed of five 
modules. The Overview module is an introduction to the CBT and module 1 
provides an introduction to the ANVIS/HUD system. Modules 2, 3 and 4 cover 
training in symbology, operation and maintenance of the ANVIS/HUD, 
respectively. Each module contains at least three lessons, several exercises and a 
test to ensure adequate coverage of the material and evaluation of the student. 


Figures 1.1 and 1.2 show a block diagram of the trainer’s structure and a page 


tad 


from a typical lesson. A typical flow through Module 1 1s highlighted in Figure 
1.1. Evident in Figure 1.2 is the interface or screen navigation controls that mimic 
the operation of a common video cassette recorder (VCR). 

A sixth module, called a refresher module, was added after completion of 
the base system (Foster and Price, 1996). This module explains key points of the 
HH-60H ANVIS/HUD computer-based trainer and incorporates video of night 
vision imagery. The refresher module is intended for use as a basic introduction 
for all rotary wing aviators in the use of the ANIVS/HUD, as refresher for those 
retuming to flight status, or to maintain currency in the ANVIS/HUD operations. 


HH-60H HUD 
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Figure 1.1. Block Diagram of the ANVIS/HUD CBT Structure 
(From Kern and Shaffer, 1996). 


astaPaa at Pata ea ara aaa es ale va elvan eee dant ete ca aata a aaaiatn ay rea Palau Seer a ewe Ferree eee ELLA ALAA SEE SSAA CAAEER: . CA SAGOEA ide Add edd CALE EAA LING Le 


Horizon Line 


my 


*-# Indicates angle of pitch and anole of 
roll. Symbol represents 0 degrees 
: angle of pitch and moves together 
<4 with Pitch Ladder symbol in roll azis. 


arete 
tere’ 
whale 
ae 
i) 





oP Me, eiteteg? 28" 

-* = 6! an) grellae 

SSeS eae”: 
= 
ba 


SS 
teeistetsetenana teens 
SENSE, 


A. 

“a 

SINS NT NY IY “ 
are ate te ote t ete ats o!e yo o's ete el etete es e's sete 018 n. 

or gitatdes statetetse calctetetarehctatsratats"e sham Aetetate, sre, ® 

: e etene%s . . nae tat atgte 2g 

26 sere eee e were ate ate" en” & 


Oa Og ate? O. Pe eaten ce eet ey ern,” ate 
os Maen "6 CeO eee ee ee 
3535 dd DRE DDD DDD DDD DDE EES 


A Range: +/- 180 degrees roll. 
= at/-45 degrees pitch 
od Pai: 3 4 | 4 For programming purposes, rolls to 
ops (j cem | | &the nght are considered positive in 
a ee | : value, while rolls to the left are 
: — ' Y negative in value. 


ote 


see tel tate ttat enact el eat atetctaneletsle totic 
SL) Saaioescotieescx 


a SAN OT ITY PW IIE TN 

TONE bntoret ctor cccretere te nee eeleN ee TON NTO OTTO OU NOON U NNT Ne TOO TOOT TTT TTT TTT TNT OTTO OTTO TT TT TT NO 

soPaerelore ctetete acetate’ cinta sterene ays ntetarane ent ater e gre evein Hug etsve avatave orstavaeste' arn tatetssarptotetnrsvtsestatalatacntelensesraveta ateceer ere a eis e°ate enecetcaters are 
Bees e g8e 2 Cage ene ne” a oie" erect oe Ole se Es #10 88 048 Me sere”? Ma" 79 Gee? eerie cle fanssatet ian Feet, rey" nave eager se, Zs 

. . ate ot oe” 4 ; A ore “6 6 eee ere . oo . ee 


OREN. AE ROT TS OTR EOL ORT OCONEE ERAT ROOT 
Fee ee 


oe ee ee ee —— —— gee eee gs =a rl e +0 
(eS ere reas Teen pele + COU o re RN Ect es t's. 
creventver sense s Serevaresero_seerseasoveec-e cent ocoreneset a nene nevelevo"b dane inversssTocavbestarerorerere"ecerevras tise scab asin entere stearate 


anata ates ate ee stole! os" a Tete! 


“4 Module: Lesson: 4 
EA Page: 4% 2f 33 


oo e. 
or aterene 

























atte” Stato aha gta "ata one a%e" eee" s"e" enero ere =f arare st arer~e eras erate arate'atgt elern'a terete Taree a ere aren taltt tere arate ersrereert re sta"e's"e"e'e e's 1 sTe"erere era lerere"elera'e era a's arene everett a a7e'e ee = Fe ergerern rare erarornte epnse ese eie tate erate ee ee eee Ae Ae ee eA eee age 
os 2 % Ofec8 eo sates on ee on . etn" ee = + tate ms eset ‘sas 2,09%8 eeu te © aseg tqdrere Ble Nic rene enn cen Sie Orn: ewes O6o ite eet © cio Saree rice nets 
‘alia A — ae Wetwos"e oe ~ al ‘= = atlalln. Tete ete” ot cot Me waste He dy de 's*o's ot oe h"s et ee ee oF ome oe eo oH AA oPe® * owebe wed. in © 2'e"e*a en’: «n'y m oN" e8y"o oMe te" oa *a’e b's o"e oho oo t'e’s ote ans © o'e'e ook i 


Figure 1.2. Page from a Typical ANVIS/HUD CBT Lesson 

(From Kern and Shaffer, 1996). 
D. PROBLEM STATEMENT 

As previously indicated, at least two years of effort, and over several 
thousand dollars have been spent in the research and development of the AN/AVS- 
7 ANVIS/HUD CBT. This effort does not guarantee that a successful learning 
tool has been produced. No matter how good the instruction content is in the 
CBT, if the student has difficulty using the computer and interacting with the 
presentation, then the value of instruction is minimized. Over time, common flaws 
associated with poor human-computer interface (HCI) design have been identified. 
Some of the most prevalent problems involve screen presentation formats and 
navigation between various screens (Kern and Shaffer, 1996). CBT applications 


often do not effectively describe system functionality to help the user understand 


and operate it. Poor design also results from a failure, on the part of the HCI, to 
reflect the form and level of user guidance considered most appropriate. If the HCI 
is poorly designed, then the potential exists for the ultimate user of the CBT to 
reject it simply because it is too difficult or too distracting from which to learn. 
To avoid this possibility it is important to determine what, if any, ease of use 
problems may exist in the ANVIS/HUD CBT program’s interface design. This 
determination relies on ascertaining and applying a method of usability evaluation 


that answers such questions as: 


© Are users able to quickly learn how to use the program? 

© Is navigation within the program easy to accomplish? 

“ Are the text based portions of the program easy to read and 
understand? 


® Are the graphics and diagrams easy to see and comprehend? 
® Does the arrangement of information on the screen make sense? 


. Does the sequence of screens make sense? 


By investigating these, and other questions, this study provides a 
preliminary indication of the success or failure of the ANVIS/HUD CBT and any 


potential improvements that may be made to the program. 


EE SUMMARY OF APPROACH 

The general approach to evaluating the usability of the ANVIS/HUD CBT 
program was to allow HH-60H aviators to interact with the system, capturing their 
reactions during and immediately after the experience. 

The Naval Aviation Safety School provided the ANVIS/HUD CBT program 
developed by Kern and Shaffer (1996) along with video recording equipment and 
administrative support. The ANVIS/HUD CBT software was installed on the 
personal computer hardware located at the HH-60H squadrons. A video camera 


was then placed near the personal computer in a position that could best capture 


both the subjects’ reactions while they worked with the CBT and a view of the 
CBT screen. The videotaped sessions were then examined to highlight any 
specific usability problems that might not otherwise be discovered. 

Immediately following the CBT training sessions, a questionnaire 
(Appendix A) was filled out by each participant. The questionnaire covered items 
specific to program navigation, appearance and on-screen instructions. Questions 
concerning the subject’s ANVIS/HUD experience and computer literacy were also 
included. The questionnaire takes approximately five minutes to complete. 

Finally, interviews were conducted with each participant. The questions and 
significant responses are detailed in Appendix D. 

Data collected from the questionnaire was used to determine general areas 
of the ANVIS/HUD CBT that required changes in presentation or structure to 
improve usability. The videotaped sessions and interviews provided assistance in 


pinpointing specific areas needing improvement. 


F. SCOPE AND LIMITATIONS 

It 1s not the intent of this research to provide an in-depth study of night 
vision goggles, night vision goggle training, learning theory or CBT development 
and use. Prior studies, specifically Kern and Shaffer (1996) and Ciavarelli, et. al, 
(1994), contain substantial coverage of these subjects. Furthermore, the 
limitations and usability problems with the current computer hardware needed to 
run the ANVIS/HUD CBT is well documented 1n these prior publications and will 
not be discussed here. The focus of this study is the ANVIS/HUD CBT program 
or more specifically, the interface provided by the software as it is run on a typical 
personal computer. Additional points that should be noted when assessing the 
scope and limitations of this project include: 


> The number of subjects available for the study is small. Only two 
squadrons fly the HH-60H. These squadrons consist primarily of 
reservists who were generally unavailable for participation. 
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e A usability lab with significant monitoring and recording capability 
was not constructed or used. All testing was done in the field with 
only available equipment. 


. Development for the actual HH-60H ANVIS/HUD is stil in 
progress. Likewise, all publications related to the system have not 
been completed. As work continues on the ANVIS/HUD system, 
inconsistencies may be identified between the CBT and the actual 
system. 


G. THESIS ORGANIZATION 

The next section of this research study is a review of selected literature 
pertinent to software usability and usability testing. Chapter III covers the 
methodology behind the data collection and Chapter IV, the analysis of the data 
collected. The remaining chapter, Chapter V, provides the conclusion and 
recommendations. The questionnaire, interview transcripts and other items 


significant to this study are attached as appendices. 


II. SOFTWARE USABILITY AND EVALUATION METHODS 


A. WHAT IS USABILITY? 

All products that we consider tools such as hammers, pens, motor vehicles 
or even computer software have certain attributes such as color or size. How easy 
that product is to use is also an attribute. One hammer may be easier to grip than 
another. A ball point pen glides more smoothly over the writing surface than does 
a quill pen. A motor vehicle with an automatic transmission is easier to drive than 
one with a manual transmission. And software that makes use of a graphical user 
interface is generally considered more pleasing to work with than a command line 
interface. Today, much greater emphasis is being placed on making products 
easier to use. 

It is important, at this point, to distinguish between a product’s functionality 
and usability. When one talks about the functionality of the product it generally 
refers to what the product can do. Usability on the other hand refers to how easily 
and quickly people can accomplish the task for which they are using a product 
(Dumas and Redish, 1993). For example, many people often consider the 
functionality of a video cassette recorder (VCR) as the ability to record certain 
television programs at a pre-selected time or to edit previously recorded videos. 
The usability of that same VCR would concern how easy it 1s to accomplish pre- 
selecting the time and program to record or the specific frames to edit on a video. 
For many people, even with the number of improvements in functionality, a VCR 
is not considered to be an easily usable product. For software, a major premise of 
usability is that a poorly designed interface may result in failure of the system to 
attain its full functional capability. Thus, in order to improve the usefulness and 
the corresponding marketability of a product many software developers are 


beginning to focus on usability. 


Usability is only a part of a system’s overall acceptability and itself has 


several attributes. Figure 2.1 shows a simple model of system acceptability and 


components, including usability and the attributes of usability. It should be noted 


that “Utility” in the figure represents functionality of the system or product. 
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Figure 2.1. A Model of the Attributes of System Acceptability 
(From Nielsen, 1993) 


When attempting improvement of a product or tool’s usability it is 


important to address each of those components that make up usability. 


The process of ensuring a usable software product generally involves two 


distinct paths, usability engineering and usability evaluation. Usability 


engineering occurs in the design phase and encompasses the inclusion of users in 


product design, adherence to design guidelines for good software, and setting 


quantitative usability goals such as specific times for task accomplishment. 


Usability evaluation on the other hand occurs at the end of the design phase. 


However, it 1s not dependent on the use of usability engineering in product design. 
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Instead, usability evaluations are best used as part of a process that focuses on 
usability throughout design and development, not as the sole time when users are 


considered (Dumas and Redish, 1993). 


B. THE PRINCIPLES OF USABILITY 

In an effort to improve software usability and remove the subjectivity from 
software usability evaluation, many usability experts have developed sets of 
guidelines or principles of usability. Usability principles, similar to the standards 
used by software developers for designing applications, help to focus attention on 
the needs of users. Guidelines, or principles can be both general or specific. A 
general guideline could be to “provide feedback” to the user. This general advice 
can be made into a more specific guideline for a graphical user interface: ensure 
that the main objects of interest to the user are visible on the screen and that their 
most important attributes are shown (Nielsen, 1993). A few authors in the 
usability field, most notably Dumas and Redish (1993), differentiate between 
usability principles as broadly defined goals and usability guidelines as specific 
measures to achieve usability goals. However, use of this differentiation is 
inconsistent and the two terms can largely be considered as interchangeable. 

Of the numerous different guideline collections that exist, some have so 
many guidelines that using them is difficult. For example, Smith and Mosier 
(1986) have developed a set of nearly 1000, Brown (1988) specifies 302 and 
Mayhew (1992) has 288. However, much of this effort can be reduced to around 
15, or fewer, short statements. Dumas and Redish (1993) present several 
shortened collections including Shneiderman’s (1992) “Eight Golden Rules of 
Dialog Design” which states that interface design should: 

Strive for consistency 


J 
Zs Enable frequent users to use shortcuts 
o Offer informative feedback 


I] 


Design dialogs to yield closure 
Offer simple error handling 
Permit easy reversal of actions 


Support internal locus of control, and 


eo TS 32 


Reduce short-term memory load 


Another list, offered by Karat, Campbell, and Fiegel, (1992), provides a 
concise compilation of what can generally be considered as the most accepted 
principles for usability design: 


Fe Use a simple and natural dialog 


a 


Provide a intuitive visual layout 
Speak the user’s language 
Minimize the user’s memory load 
Be consistent 

Provide feedback 

Provide clearly marked exits 


Provide shortcuts 


OS eo ae 


Provide good help 


—" 
oO 


Allow user customization 


—" 
—" 


Minimize the use and effects of modes 
12. Support input device continuity 
Regardless of generalization, specificity, number or label, usability 


principles provide an essential basis for creating and evaluating a usable software 


product. 


C. WHAT IS A USABILITY PROBLEM? 

1. Definition 

When considering software, usability generally refers to the human- 
computer interface (HCI). Usability attributes of a program’s interface may 


include how one moves around in the program, commonly referred to as the 


ie 


sant 


navigation metaphor, or how easy the text is to read. Other considerations may 
include consistent and easy to understand wording and appropriate graphics. 

Each of the examples of usability attributes mentioned above, and others 
such as button selection or the colors used, represents a potential usability 
problem. In broad terms a usability problem can be defined as anything that 
interferes with a user’s ability to efficiently and effectively complete tasks (Karat 
et al, 1992). More specific to software however, the definition may be stated as: 
aspects of a user interface that may cause the resulting system to have reduced 
usability for the end user (Nielsen and Mack, 1994). This directs the focus toward 
the HCI. A user can find an interface element to be problematic for many reasons. 
It might make the system harder to learn; it might make it slower for users to 
perform their tasks; it may cause usage errors; or it may simply be ugly or 
otherwise unpleasing (Nielsen and Mack, 1994). Because people’s tolerance for 
time spent learning and using tools is low (Dumas and Redish, 1993), attempting 
to find these problems early in the software development life cycle is crucial to 
user acceptance. 

Ds Determining Severity 

Determining severity of a usability problem helps to allocate more 
resources to fix the most serious problems. Such determinations can also provide 
a rough estimate of the need for additional usability efforts. Severity of a usability 
problem is largely a combination of three factors: 


lig The frequency with which the problem occurs: Is it common or 
rare? 


Ep The impact of the problem if it occurs: Will it be easy or difficult 
for users to overcome? 


Be The persistence of the problem: Is it a one time problem that users 
can overcome once they know about it or will users repeatedly be 
bothered by the problem? 


Even though severity consists of these different components, it is common 
to combine all aspects of severity in a single severity rating. An overall 
assessment of each problem can then be made to facilitate prioritizing and decision 
making. This can be accomplished using ordinal numeric scales to rank the 


problems. (Nielsen, 1996) 


D. EVALUATING USABILITY 

Over the past several years three specific methods have evolved for 
evaluating usability problems in software: heuristic evaluation, cognitive 
walkthrough and usability testing (often called empirical testing). Heuristic 
evaluation and cognitive walkthrough are methods in which a software product or 
design is inspected by individuals or groups familiar with the principles of 
usability or the software. Usability testing, unlike the other two methods, makes 
use of real users to actually test the software under controlled conditions. All 
three methods have advantages and disadvantages that will be more closely 
examined later in this discussion. Beta testing, while not a true usability 
evaluation method, is also used as an approach to finding potential problems in 
software, and as such, 1s included here in this review of evaluation methods. 

1. Heuristic Evaluation 

Heuristic evaluation is the most informal of the methods and is conducted 
simply by looking at an interface and trying to come up with an opinion about 
what is good and bad about the interface (Nielsen and Molich, 1990). This type of 
evaluation is conducted by inspecting the software interface for adherence to 
certain system design rules such as those outlined in the previous section covering 
usability principles. The principles involved in the evaluation are normally 
referred to as the heuristics and give the method its name. Generally, it is difficult 
for a single individual to find all the usability problems in a particular interface. 


Thus, evaluating an interface through the heuristic inspection method normally 
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requires between three and five evaluators (Nielsen and Mack, 1994) with each 
well versed, if not an expert, in the usability principles being applied in the 
evaluation (Nielsen, 1992). 

Heuristic evaluation is performed by having each individual evaluator 
inspect the interface alone. Only after all evaluations have been completed are the 
evaluators allowed to communicate and have their findings ageregated. This 
ensures an independent and unbiased evaluation from each evaluator. Typically, a 
heuristic evaluation session for an individual evaluator lasts one or two hours. 
During the evaluation session the evaluator goes through the interface several 
times, inspecting the various elements and comparing them to the recognized 
usability principles (Nielsen and Mack, 1994). 

a Cognitive Walkthrough 

Originally designed for the evaluation of simple Walk Up and Use 
interfaces (Lewis, Polson, Wharton and Rieman, 1990), the cognitive walkthrough 
method is meant to be used iteratively and early in a software design cycle. It is a 
task based method that serves to focus an evaluator’s attention on the user’s goals 
and actions, and whether the system supports or hinders the accomplishment of 
those goals (Wharton, Bradford, Jefferies and Franzke, 1992). Generally software 
developers or usability specialists perform a walkthrough, either as a group or 
individually. 

During a walkthrough, the steps required to accomplish a task are evaluated 
by examining how a user would interact with the interface through each step. For 
each action, the evaluators try to determine what a typical user would be trying to 
do at each point and what actions the interface allows. In evaluating each step 
necessary to perform a task, a cognitive walkthrough attempts to uncover design 
errors that would interfere with learning by exploration. The method finds 


mismatches between user’s and designer’s conceptualization of a task, poor 
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choices of wording for menu titles and button labels, and inadequate feedback 
about the consequences of an action. The procedure also uncovers implicit and 
explicit assumptions made by developers about user’s knowledge of the task and 
the interface conventions (Nielsen and Mack, 1994). 

A cognitive walkthrough normally consists of three basic phases: a 
preparation phase, an evaluation phase and a result interpretation phase. Forms, 
describing the tasks and providing detailed instructions, are used to guide the 
evaluators through the preparation and evaluation phases. The interpretation 
phase, on the other hand, is loosely structured. For example, in the preparation 
phase, the suite of tasks to be evaluated is identified and information about the 
users is noted. In the evaluation phase, questions are asked concerning each step 
within a given user task. Finally in the interpretative phase, all information 
gathered from the walkthrough is interpreted such that negative answers highlight 
steps or areas that may be difficult for the user (Wharton et al, 1992). 

3. Usability Testing 

Usability testing with real users is the most fundamental usability 
evaluation method. It provides direct information about how people use computers 
and what their exact problems are with the interface being tested (Nielsen, 1993). 
While there are wide variations in where and how a usability test 1s conducted, 
every usability test shares the following five characteristics (Dumas and Redish, 
1Oos). 


Li Each test has specific goals to meet the overall primary goal of 
improving the usability of a product. 


The participants represent real users. 
The participants do real tasks. 
What participants do and say is observed and recorded. 


A 


The data are analyzed, the real problems are diagnosed and changes 
are recommended to fix those problems. 
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Typically, in a usability test of software, specific tasks inherent to the software and 
a user population for the software are identified. For example, if one was 
conducting a usability test on a word processing program, then the task might be to 
find, open and close a file. The population of users may be office secretaries. 
From the user population several participants are recruited to work with the 
software in accomplishing the previously identified tasks. During the test the 
participants’ comments concerning the software and their performance while using 
it are recorded. Finally, the users’ opinions about the software are solicited either 
with a questionnaire or through interviews. It should be noted that what 
distinguishes a usability test from other data gathering activities such as focus 
groups, surveys and beta testing, is that the participants’ behavior while using the 
program is always noted or recorded (Dumas and Redish, 1993). In the end a 
usability test provides both quantitative data using performance measures and 
qualitative data from the observations and users’ comments. 

4. Beta Testing 

Beta testing is a simple approach to finding problems with software, though 
not an entirely reliable one. Also known as field testing, clinical tnals or user 
acceptance testing, beta testing is done by making an early version of a program 
available to users at no cost. Distribution of beta software is usually limited to 
users who have either volunteered or otherwise been selected to receive the 
product and participate in the test. No specific tasks are delineated, instead the 
users, or evaluators, are free to work with the program as they see fit. Ideally, as 
users use the program they will discover flaws, bugs and usability problems in the 
completion of their personal tasks. Discovered problems are then reported to the 
beta test director who in tur ensures that the problems are addressed in 


subsequent versions of the tested software. 
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E. WHAT METHOD IS BEST FOR EVALUATING THE ANVIS/HUD 
CBT? 


Several studies have been done that compare the various methods of 
usability evaluation. Foremost of these studies is the Karat et al (1992) study, the 
Desurvire, Kondziela and Atwood (1992) study and the Jefferies, Miller, Wharton 
and Uyeda (1991) study. 

Karat et al (1992) conducted a study comparing usability testing to 
walkthroughs. In this study, two different graphical user interface (GUI) office 
systems were evaluated to assess the reliability of the methods. In addition to 
having six individual evaluators conduct a walkthrough, they also had six pairs of 
evaluators conduct a walkthrough together to see if walkthroughs are made more 
effective when there is group interaction. The usability tests had six test 
participants using the product. The results show that the usability tests uncovered 
about twice as many problems as the group walkthroughs and three times as many 
as the individual walkthroughs. The usability tests also uncovered significantly 
more severe problems than the walkthroughs. (Dumas and Redish, 1993) 

The Desurvire et al (1992) study compared usability testing against 
cognitive walkthroughs and heuristic evaluation. Usability experts, software 
engineers and non-experts were each used for the walkthroughs and heuristic 
evaluation. The findings from this research are consistent with those of the Karat 
et al study and are displayed in Table 2.1. As shown, the usability test found more 
than twice as many problems as the expert heuristic evaluation and more than 
triple the expert cognitive walkthrough. The usability experts found about twice 
as many problems as the software engineers and even more than the non-experts 


(Dumas and Redish, 1993). 


Table 2.1. Number and Percent of Usability Problems Uncovered by Each 
Method 


Usability Test 


Heuristic Evaluation 


Cognitive Walkthrough 





(From Desurvire et al, 1992) 


In the Jefferies et al (1991) study the researchers evaluated a pre-release 
software application using heuristic evaluation, usability testing, guidelines and 
cognitive walkthrough. Specialized groups were organized around the four 
techniques to complete the evaluations: Two groups were made up of software 
engineers. One group conducted a cognitive walkthrough evaluation while the 
other applied 62 guidelines of good practice in usability as a means of evaluation. 
For the usability test, a human factors specialist conducted a usability test using 
six participants to identify usability problems. The final group, made up of user 
interface (UI) specialists, spent two weeks performing individual heuristic 
evaluations. Table 2.2 shows the total number of problems found by each 
evaluation method and the number of core problems (problems not related to the 
operating system, evaluator error or those not able to be duplicated). Table 2.3 
shows the top and bottom thirds of the problems ranked by severity while Table 
2.4 displays the mean problem severity by evaluation technique. 


Table 2.2. Total Problems Found by Evaluation Technique 
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Table 2.3. Number of Problems Found by Technique and Severity 
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(From Jefferies et al, 1991) 
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Table 2.4. Mean Problem Severity by Technique 
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(From Jefferies et al, 1991) 


Significant to the results is that none of the heuristic evaluators found more 
than 42 core problems over the two week period. The usability test however, was 
completed by the six participants in about five hours (Jefferies et al, 1991). 

These three studies allow for summary one-to-one comparisons. For proper 
inclusion, other sources are used to make the comparison of usability testing to 
beta testing. 

1. Usability Testing vs. Beta Testing 

During development of Windows 95, Microsoft Corporation made 
extensive use of beta testing. Recently however, Microsoft has demonstrated a 


shift from primarily using beta testing in favor of a more usability oriented 
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approach (Microsoft, 1997). What advantages does usability testing provide over 
beta testing? In beta testing, real users do real tasks in their actual work 
environments. However, many companies find that they get very little feedback 
from beta testers, and beta testing seldom yields useful information about usability 


problems for these reasons (Dumas and Redish, 1993): 


® The beta test site does not have to use the product. 

® Feedback is unsystematic. Users report only what they remember 
and choose to report. 

3 No one observes the beta test users and records their behavior. 

® Tasks that may be of concern may not even be tested. 


The most significant drawback of beta testing though is that it comes too 
late in the development process. At the time of beta testing the product is usually 
fully coded and difficult to change. 

Usability testing, unlike beta testing can be done throughout the design and 
development process and users can be observed and recorded as they work with 
the product or prototypes. 

Dis Usability Testing vs. Cognitive Walkthrough 

Regardless of the use of teams or individuals, cognitive walkthroughs 
generally tended to fare the poorest, as evidenced by the Karat et al (1992), 
Desurvire et al (1992) and Jefferies et al (1991) studies. This is because 
walkthroughs are a technique intended for use by software engineers who do not 
share the same viewpoint as users. Usability testing, on the other hand, involves 
real users doing real tasks. 

3. Usability Testing vs. Heuristic Evaluation 

Heuristic evaluation is the one method that is often used in place of 
usability testing. In fact a heuristic evaluation can often find more problems than 


usability testing. (Jefferies et al, 1991). However, it should also be noted that the 
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heuristic evaluation in the Jefferies et al (1991) study was done over a two week 
period by four usability interface experts. This is probably an unrealistic period 
for an evaluation. It is the type of problems found however that distinguishes 
usability testing from a heuristic evaluation. Usability testing generally finds the 
more serious and global problem (Dumas and Redish, 1993) while applying 
heuristics usually uncovers large numbers of specific, one-time (poor grammar or 
misspellings in a particular text), and low-priority problems (Jefferies et al, 1991). 
Another disadvantage of heuristic evaluations 1s a dependence upon having access 
to several people with the knowledge and experience necessary to apply the 
technique. Such people are a scarce resource and often multiple evaluators are 
necessary to obtain significant results (Jefferies et al, 1991). They may be even 
more difficult to come by if they also need to have expertise in a particular kind of 
application (Nielsen, 1992). 

4. Conclusion 

To produce a successful software product it is important to consider the 
ultimate user and potential problems they may incur while using the software in 
the everyday completion of their tasks. Without ready availability of usability 
experts nor the ANVIS/HUD CBT designers, usability testing offers the most 
effective means of evaluating the ANVIS/HUD CBT software for usability 


problems and ultimately improving the product. 


F. CONDUCTING A USABILITY TEST 

Determining the appropriate method for evaluating usability is the first step 
in a usability evaluation. Once completed the focus turns to actually carrying out 
the selected method. The use of usability testing for evaluating the ANVIS/HUD 


CBT requires careful consideration of certain aspects inherent to the method. 


tO 
tO 


hs Participants 

Typically, a usability test includes six to 12 participants divided into 
subgroups. The sub-grouping allows for identification of problems that may be 
inherent to some characteristic of the participants (Dumas and Redish, 1993). For 
example, users with significantly more computer experience may have fewer 
difficulties with a software product being tested. All participants must be 
members of the group of people who now use or who will use the product. 

Ze Deciding What To Measure 

A usability test should be designed to collect both: 


‘ Performance measures: that is, counts of actions and behaviors that 
you can see, such as: 


= time to finish a task 
_ time spent reading help files 
~ number of wrong menu choices 
= observations of frustration 
_ observations of confusion 
° Subjective measures: that is, people’s perceptions, opinions and 
judgments including: 
~ ratings of ease of learning, using or installing a product 
_ preferences over previous versions or other products 


am spontaneous comments 


Once the desired measures have been chosen then the appropriate testing and 
collection methods can be designed. 

S Usability Labs and Field Testing 

Many usability tests take place in specially equipped usability laboratories. 
However, usability labs should not be considered an absolute necessity for 
usability testing. It is possible to accomplish a usability test with no more 


equipment than a notepad. If a usability lab is desired then nearly any available 


Zo 


office space can be converted into a one. A typical usability lab will have an 
observation room and a test room separated by sound-proof, one-way mirrors, 
remote controlled video cameras and video synchronization equipment to mix 
video of the user and computer screen into a single video stream. Figure 2.2 
depicts a floor plan for a typical usability laboratory. 

Should a requirement be made that a usability test take place in the field 
then a portable lab can be devised. For those situations that do require testing 
outside of a lab, scheduling, test design and equipment setup must be carefully 


planned (Rowley, 1994). 


Camera Sound-proot wails with one-way mirrors 


focusing on 
documentation 


Camera focusing 
on the user \ 


“ 


7 












Event logger's workstation 





. Large 
Test Room a actor 
Observation duplicating 
Room user's 
screen 
User's work- 
place with Executive 
computer Observation 
and manual Lounge 
Monitors showing view 
Camera 
focusing on Extra chair for \ from each camera and Video 
computer experimenter the mix being taped editing 
inroom ora and mixing 
second user controls 


Experimenter's workstation 


Figure 2.2. Floor Plan for a Hypothetical, but Typical, Usability Laboratory 


(From Nielsen, 1993) 
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4. Pilot Test 

No usability testing should be performed without first having tried out the 
test procedure on pilot subjects. Often, one or two pilot subjects will be enough, 
but more may be needed for large tests or when the initial pilot tests show severe 
deficiencies in the test plan (Nielsen, 1993). In the case of the ANVIS/HUD CBT 
evaluation, the pilot test objective is to “debug” the equipment, materials and 
procedures. Additionally it provides the opportunity to practice the activities that 
will be done during the actual usability test. To avoid last minute adjustments the 
pilot test should be scheduled two days prior to the usability test and should: 

é Be conducted exactly as the full usability test 

® Use a test participant who represents the targeted users (Dumas and 

Redish, 1993) 

5. Having Participants Talk Out Loud During Observation 

For many usability tests, participants are asked to talk or think out loud so 
that their reactions to the product being tested can be heard. This can help 
significantly to pinpointing potential problems of products. However, because 
people do not normally think out loud while they work, they may vary in their 
ability to express themselves while they work. This in turn leads to two possible 
drawbacks. First, thinking or talking out loud may cause the participants to take 
longer in completing tasks, and second it may actually improve participants’ 
performance of the tasks. Neither drawback though, should discourage its use. 
The value of the information obtained usually outweighs the bias this procedure 
may cause. (Dumas and Redish, 1993) 

6. Interacting With Participants During Testing 

During a usability test a number of opportunities may arise that allow 
interaction between the tester and participants. The goal of the usability test will 


dictate how interactions are handled. The goal of a test on a near finished product 


2D 


may be to determine those problems that would cause a user to call for technical 
support. In that case any interactions such as assistance with the program would 
be avoided. On the other hand, the test may be on a prototype where the 
developers desire as much diagnostic information as possible. This situation may 
lead to continual dialogue between participant and tester. In general, participants 
should be allowed to do each task at their own pace in their own way (Dumas and 
Redish, 1993). Regardless of the goal there may be times when certain 
interactions are necessary. The tester may need to remind the participant to 
continue talking aloud or to move on to the next task or phase of testing. 

fe Avoiding Bias 

It is important that interactions with test participants be conducted in a way 
that avoids introducing bias. Testers can influence the way participants act and 
what participants say by biasing them with leading questions and encouraging 
either positive or negative statements. To avoid this possibility caution should be 
exercised when asking questions of participants or responding to participants’ 
questions and comments (Dumas and Redish, 1993). The interview questions 
(Appendix D) for the ANVIS/HUD CBT evaluation have been designed to avoid 
bias using guidance provided by Dumas and Redish (1993). 

8. Data Tabulation and Analysis 

A usability test generates a substantial amount of data. When the test is 
complete data may include a list of problems, quantitative data on subjective 
ratings from the questionnaire, videotapes, participants’ comments and user 
profiles. One technique available for handling such large amounts of data is 
triangulation (Dumas and Redish, 1993). The concept behind triangulation is to 
find commonalties among the various collected data which in turn may yield a 


usability problem. Figure 2.3 is a representation of triangulation. 


Problem list 


Usability problems 


Quantitative data from Participants’ comments, 
logs and questionnaires test team’s observations 


Figure 2.3. “Triangulating” - Using Multiple Sources of Data to Find the 
Usability Problems With a Product 


(From Dumas and Redish, 1994) 





Spreadsheet software is normally used in tabulating the data and calculating 
any statistics required in the analysis. Generally, only simple descriptive statistics 
and qualitative data, such as test participants’ comments, are needed to document 


the case for the presence of usability problems (Dumas and Redish, 1993). 


af 





IW. METHODOLOGY 


A. SUBJECTS 

hb Background 

Selection of participants in this research focused on potential users of the 
ANVIS/HUD CBT. This focus limited the research to the two Helicopter Combat 
Support Services (HCS) squadrons, HCS-4, located at Norfolk, Virginia and HCS- 
5, located at Point Mugu, California. Both fly the HH-60H. 

The primary missions of an HCS squadron are special warfare and combat 
search and rescue. Both missions require HCS crews to train for operations in 
hostile areas, often in the night-time environment. Special warfare involves the 
insertion and/or extraction of highly specialized teams (i.e., Navy SEALS or Army 
Rangers) in covert areas. Combat search and rescue involves rescuing downed 
aviators or stranded service members from a hostile environment or behind enemy 
lines. 

The squadrons consist of roughly 30 percent Training and Administration 
of Reserves (TAR) personnel and 70 percent Selected Reserves (SELRES) 
personnel. TARs provide the stability in the squadron by working full work 
weeks, expediting administrative matters, coordinating deployments, and 
implementing training plans. On the other hand, in order to cut government costs, 
selected reserves are required to be at the squadron for only short periods of time 
(Kern and Shaffer, 1996). As a result of this attendance policy, separating the two 
eroups for evaluation of the CBT was not attempted. 

Dis Participants 

From the available pilots in the squadrons, several were asked to participate 
in the usability testing of the ANVIS/HUD CBT. All test subjects elected to 
participate voluntarily. The final sample included seven pilots at HCS-4 and three 
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at HCS-5. No individuals with specific experience in the use of computers, 
software or the ANVIS/HUD were singled out for the purpose of including or 
excluding them from the evaluation. All participants received an overview of the 
ANVIS/HUD CBT development and an explanation of their role in the usability 
testing of the trainer. A signed written statement of consent was also obtained 
from each participant (Appendix B). The participants were asked to be candid in 
their response to the CBT and were informed that the survey, interview and video 
data would be confidential. Relevant background data regarding the survey 
participants is discussed in Chapter IV. 


B. INSTRUMENTATION 

Evaluation of the usability of the ANVIS/HUD CBT used three data 
collection methods. This approach was in keeping with the desire to gather both 
performance and subjective measures as detailed in Chapter II. 

1. Videotaping 

The Naval Aviation Safety School made available an 8mm video camera. 
Videotaping the subjects while they use the ANVIS/HUD CBT offered the dual 
benefit of: 1) capturing their reactions to the software and 2) providing a record 
for going back and verifying or retracing comments or decisions (Nielsen 1994). 
During the test the video camera was placed slightly off perpendicular to a line 
between the subjects and the computer screen. This position provided the best 
opportunity for capture of both the subjects’ reactions while they worked with the 
CBT and a view of the CBT screen. Subjects were encouraged to talk aloud 
during their session with the CBT and to point to the screen when encountering 
what they considered to be a problem with the trainer. Later examination of the 
videotaped sessions corroborated and provided detail to specific usability problems 


that might not otherwise be discovered. 


Di Questionnaire 

A questionnaire (Appendix A) was formulated based on the Questionnaire 
for User Interface Satisfaction (QUIS) (Chin, Diehl and Norman, 1988: 
Shneiderman, 1992) and an understanding of potential problems with the 
ANVIS/HUD CBT. 

The questions were organized into six groups. The first and second groups 
pertained to the participant’s familiarity with the ANVIS/HUD and computer 
experience. The third, fourth and fifth covered learning how to use the CBT, 
control and navigation of the CBT and appearance issues respectively. The last 
section consisted of a single question about time spent using the CBT. An open 
ended comments section was also included so that any concerns not covered by the 
questionnaire could be addressed. Responses to this section have been recorded 
and are included in Appendix C. 

A Likert type scale was used to indicate strength of opinion on all questions 
dealing with the usability of the CBT. Descriptive qualifiers relevant to the 
question asked were used for every answer scale and all scales had a range of 1 to 
5 which represented, respectively, low and high rating values. A Not Applicable 
(NA) option was also included. 

A pre-test of the questionnaire was conducted at NPS with both students 
familiar with the ANVIS/HUD CBT and others who were not. The students not 
familiar with the ANVIS/HUD CBT were asked to relate the questionnaire to any 
software program that they had recently used. As a result of this pre-test two 
specific changes were made: 


he Question 5.1 was modified to eliminate a potential central tendency 
error. 


Z. Differentiation was made between HUD graphics and cockpit 
diagrams 


3. Interviews 

All participants were interviewed upon completion of the questionnaire. As 
with the comments section of the questionnaire, the interviews were designed 
primarily to probe for frustration with specific parts of the CBT. Only four 
questions were asked: 


IF Did you find the portion of the CBT that you worked with to be an 
easy to use tool for learning? 


2. Was there anything that you found particularly difficult to do or 
understand? 


cy If you could change one thing about the CBT to make it easier to 
use, what would it be? 


4. Are there any features you would like to see added to the CBT? 


Significant responses to these questions are detailed in Appendix D. 


C. PROCEDURES 

A pretest walkthrough of equipment setup and instrumentation was 
completed prior to the field evaluation. The walkthrough took place in the 
Multimedia Development Lab at the Naval Aviation Safety School and reflected 
the usability test as planned for the actual testing of the CBT. Participants for this 
phase were experienced aviators though not necessarily night vision goggle users. 
During the pretest walkthrough, placement of the video camera was refined and 
any further ambiguities or typographical errors in the questionnaire or interview 
questions were corrected. 

The first “1n-the-field” usability test of the ANVIS/HUD CBT was done at 
HCS-4 in Norfolk, Virginia. The volunteer participants were given a brief 
presentation on the ANVIS/HUD CBT program and an explanation of what was 
desired from them during the test (Appendix E). It was expressly noted to each 
participant that they were not the subject of the research but were only assisting in 


the evaluation of the CBT. A statement of informed consent was then obtained 
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from each participant (Appendix B). After the presentation the subjects were 
scheduled to use the CBT so as not to interfere with their other daily tasks. 

Once the introductory presentation was finished, the CBT and video 
recording equipment were set up. A relatively secluded area in the squadron 
offices was obtained to avoid possible interruption during the sessions. 

Prior to beginning each individual session the participants were familiarized 
with the computer hardware and the location of the video equipment. The 
previous instructions were reiterated and the desire that the participants talk aloud 
during the session and point to the screen to indicate problems was stated. 
Participants were then instructed to proceed with the task of working through the 
introductory and first instructional modules of the CBT. Once the assigned task 
was finished the participants were given the questionnaire to fill out. Interviews 
were then done using the video camera to record the responses. Total time to 
complete all three portions of the test were noted so that a basis for future tests 
could be established. 


D. ANALYSIS STRATEGY 

Descriptive statistics were used to examine the data from the questionnaire. 
The number and percentage of subjects who answer a question in a specific way is 
shown in a frequency table and illustrated with a bar graph. The number of 
participants in the test was used as the base for calculating the percentages. The 
mean, median and standard deviation were also calculated for the answers to each 
question. All tabulation and calculations were accomplished using Microsoft 
Excel spreadsheet software. 

Significant comments, taken from the open ended portion of the 
questionnaire and interviews, are quoted in Appendices C and D respectively. 


Significant events from the videotapes are described in Chapter IV. 





IV. RESULTS AND DISCUSSION 


A. DATA ANALYSIS 

The findings presented in this section have been taken from the post-test 
questionnaire administered to the ten participants, interviews with the participants 
and videotapes of the usability tests. 

The responses to the questionnaire were transferred to a Microsoft Excel 
spreadsheet (Appendix F) and simple descriptive statistics were computed from the 
questions that used Likert scales. The resulting data were converted to tables and 
figures. Responses to the non-scaled questions were reviewed for any significant 
differences or similarities among the respondents. 

Like the responses to the non-scaled questions of the questionnaire, 
responses to interviews (Appendix D) were examined for trends indicating 
problems or frustrations experienced across participants. 

Finally the videotapes were viewed so that data from the questionnaire and 
interviews might be corroborated with specific occurrences. Additionally the 
videotapes served to highlight any usability problems not revealed by either of the 


other methods. 


B. FINDINGS 
1. Questionnaire 
The following pages present a detailed statistical and graphical breakdown 
of responses to each question of the questionnaire. As with the questionnaire, the 
analysis is divided into distinct subject areas. 
a. Knowledge of ANVIS/HUD 
Two questions were used to explore participants’ experience with 


the ANVIS/HUD. Most participants indicated little familianty with the 


ANVIS/HUD as shown in Figure 4.1 and Table 4.1. None of the participants in 
this study had flown with the device (Figure 4.2). 


Question 1.1: How familiar are you with the ANVIS/HUD? 
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Figure 4.1. Familiarity with ANVIS/HUD 


Table 4.1. Response Statistics for Question 1.1 


Mean 
Median 
Standard Deviation 





Question 1.2: Have you flown with the AN VIS/HUD? 
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Figure 4.2. Flight Experience with the ANVIS/HUD 


b. Participants’ Familiarity With Computers 

Participants’ use and familiarity of computers were examined as a 
potential indicator of the acceptability of a computer-based trainer. Most 
considered themselves familiar with the “Windows” operating environment 
(Figure 4.3 and Table 4.2) and said that they use a computer either frequently or 
constantly (Figure 4.4). All respondents said that they owned a home computer 
(Figure 4.5). The most popular use of a computer by participants was for word 
processing followed by online/Internet uses and databases (Figure 4.6). Finally, 
when asked to assess themselves regarding computers, eight out of the ten 


participants identified themselves as regular computer users (Figure 4.7). 


Question 2.1: Are you familiar with the “Windows” type operating system? 
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Figure 4.3. Familiarity with Windows Operating System 


Table 4.2. Response Statistics for Question 2.1 





Standard Deviation 


Question 2.2: How often do you use a computer? 
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Figure 4.4. Computer Use by Participants 


uestion 2.3: Do you own a home computer? 
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Figure 4.5. Computer Ownership by Participants 


uestion 2.4: What activities do you most often use the computer for? 
(circle all that apply) 
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Figure 4.6. Participants’ Computer Application Uses 
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Question 2.5: How would you describe yourself with respect to computers? 
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Figure 4.7. Self Description of Participants with Respect to Computers 


"i Getting started with the CBT 

This group of questions explored the participants experience with the 
CBT overview and operation instructions. Most felt that the ANVIS/HUD CBT 
was moderately easy to operate (Figure 4.8 and Table 4.3). Understanding the 
instructions however, was not considered entirely easy (Figure 4.9 and Table 4.4) 
though the terminology was generally clear (Figure 4.10 and Table 4.5). 


Question 3.1: Learning to operate the program? 
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Figure 4.8. Learning to Operate the ANVIS/HUD CBT 


Table 4.3. Response Statistics for Question 3.1 





Standard Deviation 


Question 3.2: Understanding the instructions? 
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Figure 4.9. Understanding the ANVIS/HUD CBT Instructions 


Table 4.4. Response Statistics for Question 3.2 
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Figure 4.10. Understanding ANVIS/HUD CBT Terminology 
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Standard Deviation 





dd. Navigating through the CBT 

Questions concerning navigation through the CBT received 
relatively less favorable responses. Navigation through the modules was not 
considered to be entirely clear (Figure 4.11 and Table 4.6) and the VCR type 
navigation controls were not easy to use for moving through lessons and modules 
(Figure 4.12 and Table 4.7). The next screen in a sequence was not wholly 
predictable (Figure 4.13 and Table 4.8) nor was going back to a previous screen 
felt to be easy (Figure 4.14 and Table 4.9). Determination of location within a 
task (beginning, middle, end) received the most unfavorable response of all 
questions specific to the CBT (Figure 4.15 and Table 4.10). Order of instruction 
and screens were both considered generally logical (Figures 4.16 and 4.17, Tables 


4.11 and 4.12). 


Question 4.1: Navigating through modules? 
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Figure 4.11. Module Navigation 
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Table 4.6. Response Statistics for Question 4.1 


Mean 


Median 
Standard Deviation 





Question 4.2: Use of VCR type controls for moving through lessons and 
modules? 
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Figure 4.12. Use of VCR Type Controls 


Table 4.7. Response Statistics for Question 4.2 
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Figure 4.13. Next Screen In a Sequence 
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Table 4.8. Response Statistics for Question 4.3 


Mean 


Median 
Standard Deviation 





Question 4.4: Going back to previous screen? 
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Figure 4.14. Going Back to Previous Screen 


Table 4.9. Response Statistics for Question 4.4 
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Median 
Standard Deviation 





Question 4.5: Beginning, middle and end of tasks? 
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Figure 4.15. Beginning, Middle and End of Tasks 


Table 4.10. Response Statistics for Question 4.5 
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Figure 4.16. Order of Instruction 
Table 4.11. Response Statistics for Question 4.6 


Standard Deviation 





Question 4.7: Order of screens within instructional modules? 
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Figure 4.17. Order of Screens 
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Table 4.12. Response Statistics for Question 4.7 


ee 





Median 
Standard Deviation 


é. Appearance Issues 

Participants’ responses to questions concerning the appearance of the 
CBT’s interface indicated general satisfaction. Screen size was considered about 
right (Figure 4.18 and Table 4.13) as was size of the text or font (Figure 4.19 and 
Table 4.14). Graphical displays were not hard to understand (Figure 4.20 and 
Table 4.15), however, diagrams in the CBT were felt to be less than helpful 
(Figure 4.21 and Table 4.16). Overall arrangement of information on the screen 
was considered logical (Figure 4.22 and Table 4.17). 


Question 5.1: Screen (window) size? 


a 
aq 
a 
= 
o 
fa 
a 
Sad 
Smeg 
© 
cn 
a 
= 
= 
= 
Zz. 


1 > 


Too small <——------_—----------------------__-> About right 





Figure 4.18. Screen Size 


Table 4.13. Response Statistics for Question 5.1 





Standard Deviation 
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Question 5.2: Text (font) size? 
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Figure 4.19. Text Size 


Table 4.14. Response Statistics for Question 5.2 


Mean 
Median 
Standard Deviation 
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Figure 4.20. Graphical Displays 
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Table 4.15. Response Statistics for Question 5.3 


Standard Deviation 





Question 5.4: Diagrams? 
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Figure 4.21. CBT Diagrams 
Table 4.16. Response Statistics for Question 5.4 


Mean 
Median 
Standard Deviation 
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Figure 4.22. Information Arrangement 
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Table 4.17. Response Statistics for Question 5.5 





Mean 


Median 
Standard Deviation 


ie Time Spent Using The CBT 


No participant spent more than one hour using the ANVIS/HUD 
CBT (Figire423)) 


>3 hours | 9 
2-3 hours | 9 


1-2 hours La: : 


St 
= 
Vv 
= 
° 
= 
= 
na 
= 
= 
fm 


<1 hour 


Number of responses 





Figure 4.23. Time Spent Using CBT 


g. Comments 
Comments from the questionnaire (Appendix C) were generally 
positive about the ANVIS/HUD CBT. However, four areas were mentioned more 


than once and should warrant attention from a usability standpoint. 


Ie A desire to access definitions and explanations of acronyms from 
any point in a lesson. 

z A desire to have a shortened or a more generalized version of the 
CBT. 

a A need for indication of the beginning and end of modules and 
lessons. 

4. A desire for a shorter, more concise introduction to the CBT. 
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Ds Interviews 

Similar to the analysis done on the questionnaire comments, interview 
statements were transcribed (Appendix D) and examined for multiple references to 
the same subject. In response to the first question concerning, the CBT as an easy 
to use tool for learning, one participant indicated that he had gotten lost while 
another said that the course flowchart in the overview was confusing. 

When asked if anything was particularly difficult to do or understand, two 
participants expressed frustration with not knowing when the end of a lesson had 
been reached. Two other participants felt that the CBT introduction was difficult 
and time consuming. 

The third question asked the participants to describe one change they would 
like to make the CBT. Five out of the ten participants mentioned navigation or 
page advancement issues. The desire for quick access to acronym definitions was 
expressed by two others. 

The final question inquired about any features that might be added to the 
CBT. Again access to acronym definitions was mentioned. 

St Videotapes 

Viewing the videotapes of the test sessions largely reinforces the usability 
issues established by the questionnaire and interviews. However, the videos did 
reveal other more specific usability problems. 

At the completion of some lessons the user 1s automatically advanced to he 
next lesson while other lessons return the user to the first page of the completed 
lesson. The users then became confused as to their location within the 
instructional material. This occurred repeatedly, despite the presence of the 
location index in the lower right hand portion of the interface. Other participants 
became confused over the need to return to either the main menu to begin a new 


module or the module introduction page to begin a new lesson. 
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Another problem, closely related to the previously mentioned problem 
concerned the navigation buttons. In trying to return to previous pages, 
participants were observed alternately selecting the rewind and replay buttons. 
Neither button indicated a “not available” status and the participants became 
further confused when no reaction was forthcoming from the CBT. 

The end-of-lesson exercises also represent a potential usability problem. 
The use of a green check-mark to indicate a correct answer and a red “X” as 
indication of an incorrect answer was not :mmediately obvious to many 
participants. Additionally some of the test participants seemed surprised at the 
existence of the end-of-lesson exercises. 

Along with the above mentioned usability issues several minor bugs and 
errors were discovered during the testing and are evident on the videotapes. These 
bugs, while warranting attention, do not impede use of the CBT or distract 
significantly from the learning process. 

4, Minimum System Requirements 

Previously established minimum hardware requirements for operating the 
ANVIS/HUD were found to be inadequate during the conduct of this usability test. 
Those minimums called for an IBM compatible computer with a 486SX CPU 
operating at 33Mhz, 4 Mb RAM, 17Mb available hard disk space, VGA monitor 
with 256 color capable video card and Windows 3.X. However, with the 
ANVIS/HUD CBT installed on a machine closely matching this configuration, 
albeit one that was connected to a local area network (LAN), performance was 
degraded to the point that significant waits were encountered with all actions. 
This experience indicates that the current hardware requirements are probably 
inadequate and that testing should be done on operation of the CBT in a network 


environment. 
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C. DISCUSSION 

The unfamiliarity of the participants with the ANVIS/HUD, coupled with 
their generally regular use of computers, make the ANVIS/HUD CBT an ideal 
method for instructing HH-60H pilots in the use of the ANVIS/HUD. But, as the 
findings above indicate, there are improvements that can be made to the 
ANVIS/HUD CBT that will make it a much more acceptable instructional tool. 

First, the overview module which introduces users to the CBT needs to be 
shortened and simplified. This was emphasized by both the questionnaire and the 
interviews. 

Second, the navigation metaphor used in the CBT presented problems on 
several occasions and was mentioned in every part of the usability test. The 
findings indicate that the VCR control interface is somewhat ambiguous. The play 
button suggests that the trainer might advance through the lessons automatically 
when “played”. Additionally, the difference between the rewind and replay 
buttons is not intuitively obvious and tends to confuse users even though it is 
explained in the overview. 

Third, a lack of clearly defined separations between the instructional units 
caused many participants to become lost in the instructional material. This is 
evident in the videotapes and was mentioned three times during the interviews and 
in the comments section of the questionnaire. 

Fourth, the extensive use of acronyms throughout the instructional material 
was criticized repeatedly. The present structure of the CBT expects users to read 
the definitions during the overview before moving on to the lessons. Any need to 
refer to the acronym definition list requires the user to completely exit the lesson 
with no quick method of returning to a particular point in the lesson. 

Fifth, the ANVIS/HUD CBT probably needs to be tailored to its intended 


user. Of particular note is the module covering maintenance procedures. Many 
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participants questioned the inclusion of this material and expressed a desire for a 
shortened and more pilot specific trainer. 

Finally, and perhaps most importantly, almost all participants expressed 
genuine enthusiasm for the CBT and considered their use of it to be a worthwhile 
learning experience. Most of the problems highlighted were felt to be relatively 
minor by the participants and were not significant enough to cause them not to use 


the CBT. 


V. CONCLUSION AND RECOMMENDATIONS 


A. CONCLUSION 

The design and development of the ANVIS/HUD CBT were done to fulfill 
the requirement for masters’ degrees by two Naval Postgraduate school students. 
The product of that work, while noteworthy, is still somewhat unrefined. The 
ANVIS/HUD CBT offers the Navy the promise of a portable, customized 
instructional system that could significantly enhance the Navy’s night vision 
training program. However, as with any software program, commercially 
produced or otherwise, success 1s determined not only by the functionality of the 
product, but how easy it is to use as well. Significant to ensuring ease of use is the 
proper application of recognized standards in human-computer interface design. 
Adherence to these standards during design however, does not guarantee that 
individuals who must use the ANVIS/HUD CBT will find it easy to do so. The 
results of this study serve to validate the usability testing methodology established 
in this study as a valuable way of ensuring usable software programs. Because 
usability testing bridges the gap between a finished software product and a 
potentially successful one its continued use by developers of future versions of the 
ANVIS/HUD CBT or other similar programs is strongly recommended. It 1s 
important that we do not compromise the successful adoption of new military 
training software by overlooking the needs of the ultimate user to have a simple to 


use, yet content rich tool for learning. 


B. RECOMMENDATIONS 

The results of this study and the conclusions drawn from them suggest that 
the actions listed below will enhance the ANVIS/HUD CBT and make it a much 
easier to use instructional tool: 


° Remove the Warnings and Cautions from the overview and place 
them individually at the most applicable points in the lessons. This 
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serves to shorten the overview module and to provide the necessary 
context for the warnings and cautions that presently is not available. 


Move the library of terms and acronyms to a separate section and 
incorporate hypertext links from the instructional lessons to the 
individual acronyms. This allows the user to immediately access 
definitions and explanations of any acronyms or terms used in the 
lessons. This also reduces the length of the overview module. 


Add an introductory page to the beginning of each lesson. By doing 
so, users will immediately know that they are starting a new lesson. 
The lesson introduction page should briefly explain what the lesson 
covers, the length of the lesson and number of exercise questions at 
the end of the lesson. 


Add a completion page to the end of every lesson and module. 
Similar to the lesson introduction pages, a lesson or module 
completion page serves to identify the end of an instructional section 
and provides a timely point to end an instructional session if so 
desired. 


Replace the current navigation structure with a more simplified one. 
One alternative navigation interface could use only the following 
four buttons: 


l. “Next” - advances the user to the next page. 

De “Back” - returns the user to the previous page . 

a “Go to” - allows the user to access any section (modules, 
lessons or exercises) from any page in the CBT. 

4. “Cancel” - allows the user to exit the current window or the 
program. 


Ensure that all lessons and modules automatically advance, upon 
completion, to the next lesson or module. This avoids the confusion 
that occurs when a user is returned to the beginning of a lesson or 
module from the end of that lesson or module. 


Remove the module covering maintenance procedures or build a 
second trainer that focuses specifically on maintenance 
requirements. This significantly reduces the length of the CBT and 
places the emphasis on the knowledge required by an operator. 
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C. FUTURE RESEARCH 

The recommendations offered above are an indication that future work can 
be done on the ANVIS/HUD CBT. Research into the construction of a hypertext 
linking system for the trainer is probably necessary as 1s a more optimum design of 
the navigation interface. Another area of research applicable to the ANVIS/HUD 
CBT concerns installation and performance of the software over a local area 
network (LAN). Both squadrons visited during this study were entirely 
networked. From a control and availability standpoint it may be better if the 
ANVIS/HUD CBT software could be installed on the local server and run through 
any of the squadrons personal computers (PC). This action provides a single point 
for installation of upgrades or corrections to the instructional material. 

Future research involving the usability testing methodology developed by 
this study is also advisable. By applying the data collection methods of this study 
to other software programs similar to the ANVIS/HUD, a knowledge base of 
usability factors can be developed and used in the design of future CBT systems. 
The viability of establishing a usability laboratory for that purpose should also be 


explored. 


. 
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Questionnaire for ANVIS/HUD Computer Based Trainer Users 


The ANVIS/HUD Computer Based Training software was developed at the Navy’s School of 
Aviation Safety specifically for the HCS squadrons. By completing this questionnaire you will 
greatly assist us in improving this product. The first few questions below cover your knowledge of 
the ANVIS/HUD and computers in general. The remaining questions deal exclusively with the 
trainer software and will help us to tailor it for your ease of use and learning. Finally, space is 
provided at the end of this questionnaire for your comments concerning the ANVIS/HUD CBT. 
Your time spent on this questionnaire 1s appreciated. 


Please circle the number or phrase that best reflects your answer to the question. 


1. Background 





1.1 How familiar are you with the ANVIS/HUD? 


Not at all familiar Very Familiar 
] 2 3 4 5 


1.2 Have you flown with ne ANVIS/HUD? 
No Yes 
2. Computer Usage 
2.1 Are you familiar with the “Windows” type operating system? 


Not at all familiar Very Familiar 
] 2 3 4 5 


2.2 How often do you use a computer? 

Rarely Occasionally Frequently Constantly 
2.3 Do you own a home computer? 

No Yes 


2.4 What activities do you most often use the computer for? (Circle all that apply) 


Word processing Spreadsheets 
Games Database 
Graphics Education 
Online/Internet 


2.5 How would you describe yourself with respect to computers? 


Non-user Novice Regular user Hobbyist A real geek 
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The following questions apply to your experience with the Computer based training software for 
the ANVIS/HUD. Please circle the number on the rating scale that most closely reflects your 
attitude or opinion on the statements under each section below. If you have no opinion or feel the 
statement does not apply to your experience with the trainer please circle “NA”. 


3. Getting Started 
3.1 Learning to operate the program difficult easy 


bP 2 3 4° 55 NA 


3.2 Understanding the instructions difficult easy 
1 2 3 4 5 NA 


3.3 Terminology used in the instructions confusing clear 
Wie set NE 


4. Navigation 
4.1 Navigating through modules confusing clear 
lL 2 ae 6 6UNA 





4.2 Use of “VCR” type controls difficult easy 
for moving through lessons and modules lonZ 234. SNA 
4.3 Next screen in a sequence unpredictable _ predictable 


| “2 eseeet >) OCUNNA 


4.4 Going back to previous screen impossible easy 
12 3 4 5 NA 


4.5 Beginning, middle and end of tasks confusing clearly marked 
lZeeseee | SCUCOUNA 


4.6 Order of instruction (modules) illogical logical 
) 2° 3 °4e= 5) NA 


4.7 Order of screens within instructional modules illogical logical 
] 26.3 4955 N&A 
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5. Appearance 
5.1 Screen (window) size 


5.2 text size (font) 


5.3. Graphical displays 


5.4 Diagrams 


5.6 Arrangement of information on screen 


6. How much time did you spend using the CBT? 


Less than | hour 


1-2 hours 


END 


too small about right 
2 oe ee NA 


hardtoread _—_ easy to read 
I 2 3 4 5 NA 


hard to understand _ easy to understand 
12 3 4 5 NA 


unhelpful helpful 
k 2. 3 2 SD NA 


illogical logical 
12 3 4 5 NA 


2-3 hours 3 hours or more 


Comments: (Tell us what you think of the ANVIS/HUD trainer software.) 


Copyright © 1989, 1994 
University of Maryland at College Park 
All Rights Reserved 
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APPENDIX B. RESEARCH PARTICIPANT CONSENT FORM 
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ANVIS/HUD CBT SOFTWARE USABILITY STUDY 


CONSENT TO BE A RESEARCH PARTICIPANT 


A. PURPOSE AND BACKGROUND 


LCDR Dan Rozelle, of the Naval Postgraduate School, is doing a study on the usability of 
the ANVIS/HUD Computer-Based Trainer (CBT) software. I have been asked to participate in 
this study. 


B. PROCEDURES 
If I agree to be in the study, the following will occur: 
1. Iwill use the ANVIS/HUD CBT software. 
2. If] agree, I will be videotaped using the ANVIS/HUD CBT software. 


“ 


3. Iwill be asked to respond to a questionnaire and to answer questions about my 
experience with the ANVIS/HUD CBT software. 


These procedures will be done at my place of work (HCS Squadron) and will take a total tume of 
about 2 hours. 


C. RISKS/DISCOMFORTS 


1. If, at any time, I feel uncomfortable with the procedures of this study I will be able to stop 
at any time. 


2. Iam free to decline to answer any questions I do not wish to, or to stop the interview at 
any time. 


D. BENEFITS 


There will be no direct benefit to me from participating in this study. The anticipated 
benefit of these procedures is a better understanding of improvements that may be made to enhance 
the usability of the ANVIS/HUD CBT software. 


E. CONFIDENTIALITY 
Study records will be kept as confidential as is possible. No individual identities will be 
used in any reports or publications resulting from the study. Study information will be coded and 


held by the School of Aviation Safety at the Naval Postgraduate School. Only study personnel will 
have access to the files and videotapes. 
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F. ALTERNATIVES 

I am free to choose not to participate in this study. 
G. COSTS 

There will be no costs to me as a result of taking part in this study. 
H. REIMBURSEMENT 

I will not be remmbursed for my participation in this study. 
I. QUESTIONS 

I have talked to LCDR Rozelle about this study, and have had my questions answered. If I 
have any further questions about this study, I may call him at (408) 372-2184. If I have questions 
specific to the ANVIS/HUD CBT software they should be directed to Dr. Anthony Ciavarelli of 
the School of Aviation Safety at the Naval Postgraduate School. I can contact him at DSN 878- 
2581 or Commercial (408) 656-2581. 
J. CONSENT 


] have been given a copy of this consent form to keep. 


PARTICIPATION IN RESEARCH IS VOLUNTARY. Iam free to decline to be in this study. If 
I elect to participate, 1 may withdraw from the study at any pout. 


Date Subject’s Signature 


Person Obtaining Consent 
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APPENDIX C. COMMENTS TAKEN FROM QUESTIONNAIRE 


. Excellent program. Easy to understand and a great initial 
introduction to the system 


° Overall, very informative for basic use, setup and terminology. 
Excellent for supplementing the NATOPS manual. 


° The only changes I would suggest are: 
1) Change font of letters (larger). 


2) The maintenance page/procedure page needs to be all one 
color (yellow or white). 


3) Describe what the thrust lever is. 


4) Have an icon that allows you to go directly to the acronyms 
definition page from any lesson. 


’ My only two comments concern knowing when you reach the end of 
a module. An indication that you have completed a module and need 
to return to the main menu would be an asset. Also, hyperlinks to 
acronyms and definitions would make the system easier to use. 


e Great. Ensure the setup instructions include recommendations to use 
a 486/66 or higher machine and unhook machine from any LAN it is 
on. (Note: This particular comment was directed toward slowness 
of the system experienced at HCS-4. This is addressed in the Other 
Findings sections. ) 


. Overall - good. Intro is too long and over explains many obvious 
processes. It’s better to learn as you go. Unable to confirm when 
you have completed a module. Too easy to get lost. Need feedback 
on whether a key [button] has been selected or is available 1s 
inadequate. 


2 I feel the trainer would be an invaluable tool prior to using the 
ANVIS/HUD for the first time, and then as a valuable training aid 
afterwards. The best possible situation would be to somehow have a 
mock collective stick connected to an ANVIS/HUD and a PC to 
provide the most realistic training prior to flying the HUD in the 
aircraft. Initially, I believe the HUD will be a distraction from flying 
the aircraft. It will be essential to have a syllabus designed using this 
type of computer simulation. 
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There needs to be a shortened version of this course that tells a pilot 
the essentials of what he needs to know about the control panel. 
Pilots can figure out how to mount the ANVIS/HUD and plug it in 
with one demo (a Fam 0). After that all a pilot wants to know is 
what does the control box do and how can it be operated in flight. 
You could still have the long version of the course for the detailed 
info concerning the entire system. 


The beginning tutorial needs help. Look at the current Microsoft 
Word tutorial as an example. Verbiage should be clear and concise, 
with animation to back it up rather than the long winded college 
level verbosity. 


Should be a good intro for first timers. [You] might consider a 
second unit for refresher training to avoid rehashing something not 
needed the second time. 
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APPENDIX D. INTERVIEW QUESTIONS AND SIGNIFICANT 
RESPONSES 


Question: Did you find the portion of the CBT that you worked with to be an 
easy to use tool for learning? 
Responses (HCS-4): 


a Yes, but there should be some indication that there will be exercises 
at the end of the lessons. If I would have known I wouldn’t have 
gone through the lessons quite so fast 


® Actually yes. Before using the HUD for the first time it would be 
essential. 


» Yes. The course overview had a flowchart that was initially 
confusing but I thought that overall it was pretty good. The one 
module that I worked through, I thought was fairly well laid out. 


* Not completely. Generally yes, but I got a little lost in where I was. 
Although there is a page number, unless you are really paying 
attention to where you are, you can easily get lost or skip a page. 

Responses (HCS-5): 
2 Yes, I thought it was great 


. No, it was difficult to use 


Question: Was there anything that you found particularly difficult to do or 
understand? 
Responses (HCS-4): 


e No, except for getting lost in some of the menus and not knowing 
where the lessons ended. 


© Not really, but I didn’t think that all the maintenance stuff was 
necessary for the pilots 


s The acronyms, and also I wasn’t sure whether I’d actually reached 
the end of a lesson. Maybe an end of lesson indicator would be 
helpful. 
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¥ The introduction was tedious and over-explained. In some of the 
sentences the English was weird. If you read on, it makes sense but 
if you don’t, it doesn’t make sense. 


“ The rewind and replay button. The rewind didn’t seem to work part 
of the time. And ...you can’t tell if you are doing an illegal request. 
They [the buttons] could change color or say “not available” so that 
they are not selectable. 


Responses (HCS-S): 


@ The electrical diagram was confusing, it should include more about 
what each electrical module does. 


© The course introduction was painful 


Question: If you could change one thing about the CBT to make it easier to 
use, what would it be? 


Responses (HCS-4): 


2 The control buttons. The current setup is kind of confusing. 


® Need to clear up the navigation problems so that it’s easier to 
navigate through. 
® If, when you answered a question, whether nght or wrong, it would 


automatically go to the next question instead of having to hit the 
“next question” button. I just thought that it was time consuming. 


- Maybe hyperlinks for the acronyms. 


° I don’t think there is anything that needs to be changed. There is the 
matter of figuring out the setup that is being used. This one [CBT] is 
setup like a tape recorder which isn’t hard to figure out. 


° Simply making this [the navigation panel] easier to use. Also is 
there a way to tell if you’ve done every page. There is no way that I 
can tell if I missed a page. 


Responses (HCS-5): 


° Lock the cursor (mouse arrow) to automatically go the default 
selection button. 


@ Replace the introduction with a more state-of-the-art tutorial. 


HO 


When an acronym is used in the text it should be spelled out 
immediately prior. Continuous repetition of the acronym and what 
they mean acts as a learning tool. A “Click-on” feature (hypertext) 
in the text should enable a detailed explanation or diagram of the 
selected acronym (component). 


Question: Are there any features you would like to see added to the CBT? 


Responses (HCS-4): 


It would be nice to have some indication that the next page is 
loading. (Note: this particular response is probably due to the slow 
speed of the computer being used.) 


I only used the “Play” button at the bottom. Do all the other buttons 
do something? 


Definitely the acronym thing. If there is an acronym I want to click 
on it to see what it means. 


Responses (HCS-5): 


Animation. 
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APPENDIX E. SQUADRON PRESENTATION 
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Helicopter ANVIS/HUD 
Computer-Based Training Systems 
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...to gather your reactions and opinions 
on the ANVIS/HUD CBT 


So what is the ANVIS/HUD CBT? 





qe 


Stand-alone Computer-Based Trainer 
that teaches ANVIS/HUD symbology, 
basic system operating procedures, and 

maintenance. 


¢ Designed to run on any Windows based PC 


Background 


¢ In 1992 a comprehensive study of NVG 
training was conducted 

¢ Participants included NPS, Aviation Safety 
School, NAVAIR, Naval Systems Training 
Center 


¢ Results showed the need for improved 
SOPs, revision of training doctrine and 
enhanced training equipment 





76 


Background 


¢ Multimedia Development Lab established at 
NPS under direction of Dr. Tony Ciavarelli 


¢ Several theses completed researching 
umproved NVG training 


ANVIS/HUD CBT 


¢ Developed by Shaffer and Kern in 1996 


¢ Based on system originally designed for 
UH-1 


¢ Consists of 4 training modules and an 
Overview 
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ANVIS/HUD CBT 
Overview 


¢ Describes Structure of Trainer 
¢ Demonstrates Navigation of Trainer 


¢ Library of Terms, Cautions and Warnings 


ANVIS/HUD CBT 
Module 1 


HUD System Overview 


Lesson | - Introduction to the AN VIS/HUD 
System 


Lesson 2 - Basic ANVIS/HUD Components 
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ANVIS/HUD CBT 
Module 2 


HUD Symbology Set 


Lesson | - HUD Symbology Selection 


Lesson 2 - HUD Pitch and Roll Adjustment 


ANVIS/HUD CBT 
Module 3 


HUD Operating Procedures 


Lesson | - Installing Display Equipment 
Lesson 2 - Starting the HUD System 
Lesson 3 - Using the HUD System 
Lesson 4 - Egress Procedures 
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ANVIS/HUD CBT 
Module 4 


HUD Operating Procedures 


Lesson 1 - HUD O-Level Maintenance 


Lesson 2 - Electro-Static-Discharge 
Considerations 


Lesson 3 - HUD Unit Descriptions and 


Maintenance Procedures 


ANVIS/HUD CBT 
Module 4 (cont.) 


HUD Operating Procedures 


Lesson 4 - HUD Receipt, Shipment, and 


Storage Procedures 


Lesson 5 - HUD Operational Checks 


Lesson 6 -Fault Isolation and Troubleshooting 
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...to gathering your reactions and 
opinions on the ANVIS/HUD CBT 





Usability Testing 


¢ Done by most major software vendors 
¢ Focus is on usability vice functionality 


¢ Attempts to answer questions such as... 


Are users able to quickly learn how to use the 
program? 


Is navigation within the program easy to 
accomplish? 


Is the text easy to read and understand? 
Graphics and diagrams? 
Arrangement of information? 


Sequence of screens? 





Usability Testing 


e Volunteers are needed to... 


— use the CBT (Overview and Module 1) while 
being videotaped 

— fill out a brief questionnaire 

— answer a few short questions 





Questions? 
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APPENDIX F. SPREADSHEET DATA FOR THE ANVIS/HUD CBT 
USABILITY TEST QUESTIONNAIRE 
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